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ABSTRACT 

Results froa classical power analysis (Brewer, 1972) 
suggest that a researcher should not set asp (when p is less than a) 
in a posteriori fashion when a study yields statistically significant 
results because of a resulting decrease in power* The purpose of the 
present report is to use Bayesian theory in exasining the validity of 
this generalization. Osing the t'-test and Bayes' theorea we show that 
while the substitution of p f or a (when p is less than a) reduces 
unconditional or sisple power, it actually increases the conditional 
power of the test— i.e., the probability that the alternative 
hypothesis is true given a statistically significant result. Because 
^f this, the substitution of p for a a posteriori (when p is less 
than a), i.e., acting as though the value of a is egual to the 
obtained p value, seeas to be a haraless practice at worst, and on 
the positive side, provides iaportant inforaation to the research 
consuaer interested in the validity of the alternative hypothesis 
given the data of the study. (Author/BC) 



The fiffeot of Substituting p for alpha On The Uheoaditioaal aad 
C«iditional Powers of A Null Uypott^sis Test 

Abstract 

Besults troA classical power analysis (firewer, 1972) suggest that a 
researcher should not set a " p (wl^ P < o) in an a i»8teriori fashicm when 
a study yields statistically significant results because of a resulting de- 
crease in power* The purpose of the preset report is to t»e Bayesian theory 
in exanining the validity of this generalissation* Using the t test and Bayes * 
theorea we show that while the substitution of p for a (^hen p < a) redueec* 
uiMonditional or simple power« it actually incr^tses the conditional power of 
the test— i«e*« the probability that is tn» given a statistically signi- 
ficant result* Because of this« the substitution of p for a a posteriori 
(when p < a)> i«e«9 acting as tiu>ugh the valt» of a is equal to the obtained 
p value I seeas to be a haraless practice at worsts and the positive sidet 
provides inportant inf oraaticm to tt» res^rch ^msiaier interested in tt» 
validity of the alternative hypothesis (H^) eXv&a iha data of the study* 



Oi DEPARTMEMTOFNeALTM. 
eOUCATIOM A (VSLPAHE 
•lATIOWAL IMITiTUTE OP 
eOUCATKM 

S DOCUVFNT MAi BFEm 8EPR0 
DuffD FXACUV as RF''£ivCD FROU* 
•h6 PEBSOfg OB OSGAMJATIOIW OR'GiN 
i' Vr, IT POINTS Of ^ifA OS OPiNlONi 

• '-'■fo 00 twOT "wFcEiSASiLT »£r«e 

■ ( NTOff IClAl. NAT'ONA,. .mSTltl.,Ti Of 



The Effect of Substituting p for alpha On The Unconditional and 
Conditional Powers of a Null Hypothesis Test 

February 24, 1974 

Recently, a good deal of attention has been focused on the area of power 
analysis in the behavioral sciences. Cohen (196^). Overall (1969), Tversky 
and Kahnnan (1971), and Brewer (1972, 1973) among others, have strongly sug- 
gested that the power (the probability of rejecting the null hypothesis (H^) 
when the alternative hypothesis (Hi) is true) of a statistical test should be 
determined prior to conducting an experiment. The importance of this analysis 
is suggested by Tversky and Kahnman (1971): (l)the results of a power analysis 
may Indicate that the study should not be conducted -jnless the sample size is 
substantially Increased; (2)power analysis is necessary to the explanation 
of negative results; (3)the results of power analysis provide an indication 
of the probability of correctly rejecting H^* 

Cohen's concern with power analysis in the behavioral sciences has resulted 
in the first easily manageable set of procedures and tables for the calc Tatlon 
of the power of statistical tests. The use of Cohen's tables to determine the 
power of a test requires the researcher to combine a number of readily avail- 
able pieces of information: (l)sample size, (2) alevel, and (3)effect size 
(ES) which Cohen has defined as "the degree to which the phenomenon is present 
in the population." 

As part of this trend. Brewer (1973) has conducted an extensive analysis 
of the statistical tests appearing in several educational and psychological 
research journals during the last few years and found that the power In many 
studies was inadequate. Brewer (1972) has discussed a nuu^er of procedures 
for maximizing the power of statistical tests used in behavioral science re- 
search. One of these is simply to avoid the "practice of taking a sample. 
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calculating p (the probability of the statistic under H^) and then equating 
this value to a (when p< o).'* He contends that the minliaum sample size 
necessary to detect a significant departure from the null condition at a" .05 
1.8 altogether too small to detect the same departure at a a.Ol because change 
In the size of a is directly related to change in the power of the test. He 
provides the following as an example: 

If a Jt test is conducted with n^ n2 * 50 
and medium ES, then power is approximately 
.70 for o= .05 and .45 for a* .01 ... 
It is clear that if power is to be maintained 
at .70 for this senile and ES, then the test 
is not significant at the .01 level even if 
p < .01. The fact that p .01 only allows 
the researchers to reject at the ot » .05 
level under these conditions. (Researchers 
may, of course, equate p and a if they are 
willing to take the resulting reduction in 
power.) (p. 395) 

Here, Brewer argues against substituting p for a (when p < a) strictly 
because the adoption of such a decision rule reduces the power of the test. 
While others (e.g., Cohen, 1973; Dayton, et al, 1973) have pointed out that 
the power argument advanced by Brewer is inappropriate within the classical 
(non-Bayesian) framework, a demcmstration of the effects of such a policy 
on the conditional power of a null hypothesis test seems useful at this time. 
In the following discussion, Bayes* theorem and the t test are used for this 
purpose . 

A Bayesian Perspective 
Overall (1969) has noted that particular concepts fvoa Bayesian theory 
have logical implications for the design of experiments. Bayes theorem 
provides a formal statement of the probability of the null hypothesis being 
true given a statistically significant result, and may be stated as; 
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where PCHq | D) Is the probability of the being true given a statistically 
significant result (D)| P(D | Hq) is the probability o£ significant result 
given a true null hypothesis; P(H^) is the a priori probability that the 
null hypothesis is tme and P(D) is the probability of a statistically 
significant result ^rtien either the null or alternative hypothesis is trtie. 
Overall defines a conditional alpha probability, P(Ho j as P^^®" 
bablllty that a result which is judged statistically significant has oc- 
curred by chance rejection of a valid null hypothesis." He Illustrates this 
difference between the simple and conditional a levels by means of the follow- 
ing example: 

Consider the case of an 1- .istigator who works 
entirely within the area wnere the null hypothe- 
sis is always true. If consistently employs 
the proper classical statistical model with 
o » ,05 or a «* .01, the probability will be only 
.05 or .01 that he will reject a null hypothesis 
when it Is in fact valid; hoiraver, of the statis- 
tically significant results reported by this hap- 
less investigator, lOCf% will be due to chance. 
This ad absurdum example illustrates the differ- 
ence between simple alpha probabilities and con- 
ditional alpha probabilities... (p. 286) 

What is the practical significance of conditional alpha probability? 

To answer this question. Overall cites a case «^lch he claims exemplifies 

psychiatric drug research. Typically, this area is characterized by small 

saaqple size, low power tests, and unlikely treatment differences. Given 

these conditions and the simple a level of .05, the conditional eilp^|^p« 

level was calculated to be .643. Overall concludes that the analysis 



Indicates that approximately two-thirds of the statistically significant 

results will be type I errors! 

In light of Overall's analysis, an exantination of the ramifications of 

substituting p for a seems in order. For while substituting p for a (when 

p < a) clearly reduces unconditional or simple power, ^ this particular notion 

of powet is irrelevant once the data have been inspected. However, the effect 

of this action on conditional power (i.e., the probability that Hj^ is true 

given a statistically significant result) seems appropriate and is the focus 

of the remainder of this discussion. 

In terms of Bayes' theorem, conditional power is defined as: 

P(DIH,) P(H,) 
^ ^^lP> ' p'(D) ' 

where P(H, |D) is the probability of a true alternative hypothesis given a statis 
tically significant result (D); P(Hj^) is the a priori probability that the alter 
native hypothesis is true; P(DjH^) is the probability of a significant rosult 
(D) given that Hj^ is true; P(D) is the probability of a statistically signifi- 
cant result when either or is true— i.e., P(D|Hj) P(H^) + P0)|H^) P(H^). 

We are now in the position to deossnstrate the effects of substituting p 
for a (when p < a ) on the conditional and unconditional power of a statistical 
test. The variables of interest are: 

a. Effect Size (ES) - the degree to which the phenomenon exists 
(Cohen, 1969). Cohen^s small (.20), medium (.50) and large 
(.80) ES were used. 

fe- Sample Size - small (n « 20), medium (n « 50!) and large 
(n « 100) sample ^izes were used. 

c, P(Ho) - the a priori probability of the null hypothesis 
being true: "small (.20), medium (.50), and large (.90) 
prior probabilities were used. 



^ It is recognized that the term "unconditional or sinq>le" power is not 
entirely appropriate since power is always conditional on a number of 
factors. However, the term as used here refers to power in the classical 
sense. 
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d. Alpha Level - .05 was selected. 

e. p-level - .01 was selected. 

£. Statistical Test - Student's t test was selected. 

For each con&ination o£ effect size, sai^le size, P(H^)i alpha level 
and p-level, the conditional alpha probability PCH^|d) and the conditional 
power probability PCH^|d) were calculated. The results of these calculations 
are shown in Table 1. Notice that for a given ES, sample size, and P(H^)t 
the unconditional or single power probability of the t test decreases when 
single a is reduced from .05 to .01 %Aiile the conditional power probability 
of the test actually increases . 

While attention is still focused on the table, two other relationships 
nay be noted: 

a. As the effect size (ES) increases, the probabaity of a valid 

null hypothesis given a significant result [P(H^|D)] decreases 

at each level of saa^le size and each level of the probability 

of a valid H [PCH ) 1 whereas the probability of a valid alter- 
o o 

native hypothesis given a significant result IpCHj^|D) j increases 
at. each level of sample size and P(R^). 

b. As sample size increases, the probability of a null hypothesis 
given a significant result IpCH^Id)} decreases at each level of 
effect size (ES) and each level of the probability of a valid 

lP(H^) ) whereas the probability of a valid alternative 

hypothesis given the data [P(Bj^|d)] increases at each level 

of ES «ui P(H ). 

o 

Conclusions 

It is clear that if p is substituted for a i%Aiea p < a) a posteriori, 
then power in the classical sense is reduced. But, clearly, this imconditional 



power Is no longer relevant once the data have been Inspected. Following the 
notions of Overall and others (e.g., Roseboom, 1960), one must conclude that 
if power must be calculated, only conditional power is relevant, 

Fran the Bayeslan perspective, substituting p for a (when p < a) has no 
adverse effect on the measure of power and, on the positive side, provides the 
consumer Is interested in the tenability of given the data with useful 
information. 



References 



Brewer, J. K. On the power of scatlstical tests. American Educational 

Research Journal . 1972, £(3), 391-401. 
Brewer, J. K. & Owen, P. W. A note on the power of statistical tests. 

Journal of Educational Measurement . 1973, 10(1), 71-74. 
Cohen, J. Statistical pw^r analysis for the behavioral sciences . New York: 

Academic Press, 1969. 
Cohen, J. Statistical power analysis and research results. Aa»rtcan 

Educational Research Journal . 1973, 10(3). 225-23C. 
Dayton, C. M., Schafer, W. D. & Rogers, B. C. On appropriate uses and inter- 
pretations of power analysis: A cooment. American Educational 

Research Journal . 1973, 10(3), 231-234. 
Overall, J. E. Classical statistical hypothesis testing within the context 

of Hayes Ian theory. Psychological Bulletin . 1969, 71, 285-292. 
Rozebooffl, W. W. The fallacy of the null- hypothesis significance test. 

Psychological Bulletin . 1960, 57, 416-428. 
Tversky, A. & Kahnnan, D. Belief in the law of small ntsatbers. ftivchological 

Bulletin . 1971, 76, 105-110. 



Table 1 

Conditional alpha probability [P(H^|D)] and conditional po« 
for selected values of effect size, saaqple sise, a priori prob. 

being true [P(H )], alpha level and p 





Siaall Effect Size (.20) 


Medium Effect Size (.. 


Small Sanqtle 
Size (20) 


Medium San^Ie 
Size (50 


Large Sample 
Size (100) 


Small Sample 
Size (20) 


Medivtm Soople 
Size (50) 




P(F^) 


P(H^) 


P(H^) 


P(H„) 


P(H^) 




Small (.20) 


Medium (.50) | 


Large (.90) 


Small (.20) 


Medium (.50)^ 


Large (.90) 


Small (.20) 


Medium (.50) 


Large (.90) 


Small (.20) 


Medium (.50) 


Large (.90) 


Small (.20) 


Medium (.50) 


Large (.90) 




1 Unconditional 
a - .05 


P(HJD) 


.12 


.36 


.83 


.07 


.23 


.73 


.04 


.15 


.61 


.04 


.13 


.58 


.02 


.07 


.39 




P(HjD) 


.88 


.64 


.17 


.93 


.77 


.27 


.96 


.85 


.39 


.96 


.87 


.42 


.98 


.93 


.61 




P(D|Hj) 


.09 


.09 


.09 


.17 


.17 


.17 


.29 


.29 


.29 


.33 


.33 


.33 


.70 


.70 


.70 




Unconditional 
p - .01 


PCH^jD) 


.11 


.33 


.82 


.04 


.14 


.60 


.02 


.08 


.43 


.02 


,07 


.39 


.00 


.02 


.17 




P(H,|D) 


.89 


.67 


.18 


.96 


.86 


.40 


.98 


.92 


.57 


.98 


.93 


.61 


1.00 


.98 


.83 


1 


P(D|Hj) 


.02 


.02 


.02 


.06 


.06 


.06 


.12 


.12 


.12 


.14 


.14 


.14 


.45 


.45 


.45 
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Table 1 

*>illty [P(H^|D)3 and conditional power probability IPCHj|D)] 

et sise, sample size, a priori probability of the null hypothesis 

Ing true (P(H^)], alpha level and p - level 





Medium Effect Size (.50) 


Large Effect Size (.80) 


e 


Small Sample 
Size (20) 


Medivnn Saople 
Size (50) 


Large Sample 
Size (100) 


Small Sairr^le 
Size (20) 


Medium Sas^le 
Size (50) 


Larg(<! San^le 
Size (100) 




P(H^) 


P(H^) 


Pm ) 


P(H^) 


P(HJ 


P(H„) 




o 

• 

1 


Medium (.50) 


Large (.90) 


Small (.20) | 


Medium (.50)1 


Large (.90) 


Small (.20) 


o 
• 

§ 


Large (.90) 


Small (.20) 


Medium (.50) 


Large (.90) 


Small (.20) 


Medium (.50) 


Large (.90) 


Small (.20) 


O 

• 

i 

•o 

SB 






.04 


.13 


.58 


.02 


.07 


.39 


.01 


.05 


.32 


.02 


.07 


.39 


.01 


.05 


.31 


.01 


.05 


.31 




.96 


.87 


.42 


.98 


.93 


.61 


.99 


.95 


.68 


.98 


.93 


.61 


.99 


.85 


.69 


.99 


.95 


.69 




.33 


.33 


.33 


.70 


.70 


.70 


.94 


.94 


.94 


.69 


.69 


.69 


.98 


.98 


.98 


1. 00 


1.00 


1. 00 




.02 


.07 


.39 


.00 


.02 


.17 


.00 


.01 


.10 


.00 


.02 


.17 


.00 


.01 


.09 


.00 


.00 


.08 




.98 


.93 


.61 


1.00 


.98 


.83 


1.00 


.99 


.90 


1.00 


.98 


.83 


1.00 


.99 


.91 


1.00 


1.00 


.92 




.14 


.14 


.14 


.45 


.45 


.45 


.82 


.82 


.82 


.44 


.44 




.91 


.n 


.91 


1.00 


1.00 


1.00 
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